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ABSTRACT 



This study investigated whether changes occur frequently in 
the ratings of colleges by "U.S. News and World Report." The annual rankings 
of colleges receive considerable publicity and are taken quite seriously by 
the media and the public. The three lowest tiers of colleges are not ranked. 
The study was intended to determine how much movement occurs in those 
ratings, whether rising or falling. Ratings for 6 years (1996 through 2001) 
were studied for 162 nationally ranked 4-year liberal arts colleges, and the 
study was replicated with colleges from Southern Region that offered master's 
level programs. At the national level, change occurred in 14% of the 
opportunities, so that most of the year-to-year comparisons showed no change. 
There were very few institutions that appeared less than six times in the 
rankings. At ■ the regional level, change occurred in 19% of the opportunities. 
In general, there was sufficient stability within the rankings to make one 
take a skeptical view of efforts to improve an institution’s rankings. At the 
same time, the findings bode well for those institutions that are satisfied 
with their rankings. (Contains 9 figures and 13 references.) (SLD) 
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BEST COPY AVAILABLE 



Each year the U. S. News and World Report publishes rankings of colleges and 
universities in the United States. As a guide to college applicants and their parents, these 
rankings, it is believed, are taken quite seriously. Decisions affecdng innumerable academic 
careers are at stake; from the standpoint of compeddve higher educadon insdtudons, these 
stakes are high. Since colleges and universides recognize the impact that these published 
rankings (and many others) can have, the leaders of many of these insdtudons are eager to 
improve their standing. Thus, presidents, admissions officers and others pay pardcularly 
close attendon to the rankings and the criteria that influence them, undoubtedly aspiring to 
influence their own insdtudons in a way that will have a posidve impact on the radngs and 
the benefits that will follow. 

Perspectives from the Literature 

Rankings of academic quality have been a part of the U.S. academic scene for 
approximately 100 years (Webster, 1992). Today, nadonal rankings have become widely read 
and fundamentally a big business 1 thanks in large part to U. S. News & World Report, which 
began publishing reputadonal rankings of U.S. colleges based on a survey of college 
presidents in 1983 (McDonough, Antonio, Walpole, & Perez, 1997) and has become the 
“gold standard” of the ranking business (Ehrenberg, 2001). Furthermore, the influence of 
the USNWR and other rankings is likely to increase because of their ready availability on the 
Internet (Webster, 2001). 



1 McDonough et al. (1997) estimate that total revenue from the various newsmagazines that publish college 
rankings and guidebooks reach $16,000,000 a year in sales alone. 
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USNWR ranking has long been the subject of widespread criticism for their use of 
“soft data,” use of questionable or (some say) meaningless criteria, arbitrary weighting 
schemes, and technical difficulties (e.g., Gottlieb, 1999; Thompson, 2000; Webster, 2001; 
Wright, 1992). In fact, the editors of USNWR have gone out of their way to change their 
methodology on an annual basis in order to mollify their college critics (Machung, 1995; 
McDonough et al., 1997, Morse, 1995). 

Arguably, a major reason why U.S. News & World Report rankings have become so 
immensely popular and influential is not because they have been useful for college choice 
but primarily because colleges and universities have aggressively used them to promote 
themselves (Machung, 1998). 

The tier rankings are important for college and university administrators because 
they partly define the institution's market niche, influence the perception of the institution by 
prospective students, which affects enrollments and operating budgets, and serve as a guide 
to the institution's strategic planning (Webster, 2001). 

The January 1997 issue of State Policy Reports highlighted a relationship between 
school quality ratings and the return on investment that states receive for their support of 
public higher education, thus making the suggestion to tie college funding to ranking status 
(Hovey, 1997). 

Monks and Ehrenberg (1999) analyzed the changes in USNWR ranking for the top 
national universities and liberal arts colleges and found that “a change in rank does have a 
significant influence on admissions outcomes and institutional pricing decisions . . . these in 
turn will have an impact on the institutions’ future rank” (p. 10). Similarly, Bednowitz (2000) 
found changes in USNWR ranks are highly related to admissions outcomes at MBA 
programs. 
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One might ask, “Why study changes in ranks?” One perspective is that, “ . . . since 
we may change our methodology from year to year, we do not invite readers to track 
colleges' annual moves in the rankings” (Morse & Flanigan, 2001, online). Overall, 60% of 
students find newsmagazine college rankings to be not at all important, 30% cite rankings as 
somewhat important, and 11% rate them as very important in their college choices 
(McDonough et al., 1997). Studies find that rankings are primarily used by more well off 
students from college-educated families. The students who do not have schools with good 
guidance operations or knowledge about college from their parents are not using the 
rankings newsmagazines in their college choice decision-making (McDonough et al., 1997). 

Alternatively, the rankings issues of the newsmagazines sell. Ehrenberg (2001) 
keenly noted that “ ... it is the change in the numerical rankings of institutions near the top 
of each institutional category, as well as the changes in the quartile rankings of some lower 
ranked institution from year to year that sells lots of copies of [USNWR] magazines” (p. 1). 
Bednowitz (2000) in his study of the impact of USNWR rankings on the Business Schools 
found that the recruiting choices of employers are not statistically related to a programs 
ranking for a given year. Rather, employers react exclusively to changes over time. 
Bednowitz further concludes that “ . . . students may not be behaving in their best interest 
by reacting to single-year changes in rank . . . Based on employers’ behavior, it makes more 
sense for a student to take a longer term view and attend a school which has been highly 
ranked for many years . . ., even if that school is not currently ranked highly” (p. 31). 
Ehrenberg (2001) finds that colleges’ shifts in the rankings are a reflection of the 
competitiveness of the market segment where the institutions operate. 
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The Issue Addressed by the Current Study 



Some would argue that such efforts should not be equated with direct improvements 
in institutions. Others might argue that there are overall benefits that stem from the efforts 
of U. S. News and World Report , and dozens of others, to perform a service for consumers. 
This paper does not deal directly with the issue of whether the time spent paying attention to 
college rankings is well spent in the sense that they lead directly to institutional 
improvements. However, the paper deals with the related issue of whether such efforts are 
well spent in the sense that they are likely to pay off in reaching the immediate goal of higher 
rankings. To state the issue another way, we asked the following questions: How often do 
changes in rankings occur? Do they frequently tend to improve? Are positive movements 
in rankings as likely as negative movements? This study is not intended to discourage those 
efforts but rather to provide an accurate picture of how much movement, whether rising or 

falling, has occurred historically. Such information is the best guide on how much 

* 

movement, positive or negative, to expect in the future. It was hoped that the answers to 
these and related questions would help institutions to know whether their efforts to improve 
their rankings are likely to succeed. 

Method 

This part of the report summarizes the methods used as applied to the national 
liberal arts category of institutions. As will become evident, the replication study (involving 
Southern Regional Universities, to be described below) used very similar methods in order to 
insure that comparisons could be made between the two studies. 
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At the outset, we undertook a study of the rankings for 6 previous years in the US. 
News and World Report for nationally ranked four-year liberal arts colleges. Six years of 
rankings (1996 through 2001) gave the possibility of movement over five years: 1996-97, 
1997-98, 1998-99, 1999-2000 and 2000-2001. Colleges could stay the same over all four years 
or change. If they changed, they could change up or down repeatedly in several possible 
combinations. (Note: the years named— 1996-2001— or years of reference are dated one year 
after the date of the actual issues in which the rankings appear.) 

Although the 2002 rankings were available well in advance of the 2001 ASHE 
national conference, we decided not to include those results for the liberal arts colleges’ part 
of the study. Examination of the 2002 results revealed that a radical shift in methodology 
had occurred. Part of the change was the addition of dozens of institutions that had 
previously participated in other categories of U.S. News rankings. Our impression was that 
formerly distinct categories of institutions had been conflated in the new rankings. Thus, it 
was our impression that the new data for liberal arts institutions warranted a study in its own 
right. We have not had the leisure to undertake such a study. However, as a “work in 
progress,” in the near future our study will extend into the special circumstances and 
findings associated with the 2002 rankings of liberal arts colleges. 

The method used involved entering the names of all colleges and their tier 
assignments (1-4) for each year into a spreadsheet. For tier one assignments only, we made 
entries of the unique institutional rank (1 -#, where n was the lowest rank), as published by 
U. J’. News and World Report . Sorting the spreadsheet by college and by year within colleges 
afforded an opportunity to identify cross-tier movements. Most of the findings reported 
below stem from these simple steps and the various counts and other calculations that they 
made possible. 
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Results and Discussion - National Liberal Arts Colleges 



If one looks at the probability of any changes taking place out of all opportunities for 
change to have occurred, that value turned out to be .14. In other words, change occurred 
in 14 percent of the opportunities. Therefore most of the year-to-year comparisons showed 
no change. The probability of no change from one year to the next was 1.00 minus .14 or 
.86, i.e., 86 percent of the opportunities. Further, the probability of a change (.14) was 
nearly evenly divided between rising and falling, with the probability of reaching a higher tier 
in the next being approximately .08 and that of declining being approximately .07. 

Looking at the institutions, there were 162 represented over the 6 years. That is only 
slightly higher than the 160 that would be expected if there were 40 in each tier each year 
and there were no newcomers. In fact, there were several, but only very few, that appeared 
less than 6 times. The breakdown is shown in Figure 1. 

The obvious implication is that, once included among the "best liberal arts colleges," 
an institution is likely to stay there. The other side of this positive conclusion is that 
movement within the 160 or so best is extremely slow and difficult. There were 91 
institutions that showed no movement at all over 6 appearances. This conservatism is shown 
most of all in tier 1, which is partly due to the "ceiling" effect; simply put, tier 1 institutions 
cannot rise to a higher tier. However, tier 4 membership is also conservative, even though 
the institutions making up that tier are at most risk of falling off of the "best colleges" list 
completely. 

Since the bottom 3 tiers are not ranked, it is difficult to determine the average 




movement per institution over the 4 years of possible change. Such study fell outside the 
resources of time available for the current study. Fortunately, there was a shortcut to 
making a relevant estimate. The shortcut was to calculate the average change in rankings 
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within tier 1 between 1996 and 2000. The average change over four years of the 40 or so 
institutions in her 1 turned out to be 1.94 posidons. (Note: This result was calculated for 
1996-2000 and has not been updated with 2001 data.) 

A finding of 1.94 posidons is interesting because it represents only 5 percent of the 
range between ranks of 1 and 40. Inspection of the rankings from year to year showed that 
they were very tightly confined, going up or down at most only a few places. Two colleges 
with the greatest upward movements, from the 1996 to the 2000 rankings, moved up 10 
places. A third college moved up 7 positions. 

If movement happened similarly within the other three (unranked) tiers, we may 
expect that movement between tiers might be fairly rare. However, when considered over 
all 6 years, movement up or down was not particularly rare. Considering the movement 
from tier 4 to tier 3, there were 49 institutions that had the opportunity to make that 
movement during at least one year in the period 1996-2001. Of these 49 institutions, there 
were 19 that did move up at least once. This number was equivalent to approximately 39 
percent. It should be borne in mind that this rate is fairly low on a per year basis. Over six 
years (five transitions), the average per year upward shift was slightly less than 8 percent. 

However, even if movement was not particularly rare, one needs to know whether 
the overall upward movement tends to be balanced by downward movements. In other 
words, can the overall movement pattern be described as “ zero sum,” with upward 
movement of some colleges balanced and compensated for by the downward movement of 
other or (in many cases) the same colleges. One relevant observation was the following. 
While 19 colleges moved from tier 4 to tier 3, 19 also made the reverse trip from tier 3 to tier 
4. 
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There were 62 institutions in tier 3 that had the opportunity to move up to tier 2 
between 1996 and 2001. There were 27 that actually moved up. Thus, again such movement 
was not unusual. The percentage of upward movers over all six years was 43.5. On a per 
year basis, that is slightly more than 8 percent per year. There were also 28 that moved 
down from tier 2 to tier 3, suggesting again a “zero sum” situation. Again, many of the 
same colleges that moved up also moved down. 

The number of institutions that had multiple movements (both up and down) was 
examined. Of the 62 institutions that experienced any transitions from one tier to another, 
those transitions could be divided into four categories. The first category consisted of 
simple reversals; that is, the ranking went up in one year and down in a subsequent year, or 
vice versa. There were 25 (40.3 percent of the “movers”) that fell into that category. The 
second category was of institutions with 1 Vi transitions; i.e., there was a shift (up or down) 
that was cancelled out by the reverse shift and then reinstated by a third shift back to the 
original tier. There were 8 transitions falling into that category (12.9 percent). Third, there 
was one “double reversal,” consisting of a change from tier 3 to 2 and back to 3, followed by 
a shift to 4 and back to 3. Finally, there were two special cases. One institution was 
assigned to tier 4, although it had refused to provide data to U.S. News. Subsequendy the 
institution rose to tier 1, and still later fell to tier 2. Finally, one institution experienced a 
“double slide,” from 2 to 3 and then from 3 to 4. In all, 36 of the 62 institutions (or 58.1 
percent) had multiple movements within the 6 years. 

Another related question is how much the institutions that rose (or fell) in their tier 
assignments actually changed in rank relative to other institutions. Since U.S. News does not 
publish detailed ranks except within tier 1, the existing data do not permit an easy answer to 
that question. However, we may turn again to tier 1 to find the answer to an analogous 



question at that level. (The following test was done for 1996-2000 data and has not been 
updated for 1996-2001.) To address this question, we created an artificial division within tier 
1 between institutions with a rank greater than 20 (i.e. 21-40) and those with a rank less than 
or equal to 20 (i.e. 1-20). Let us call them tier 1A (most premier) and tier IB (less premier). If 
we look for movement between the two new groups, we can measure the change in rank of 
those colleges that moved in either direction. It turns out there were only two institutions 
that rose in rank. One moved from rank 21 to 11 and another rose from 23 to 18. There 
were two other close calls: one moved from 34 to 24, thus approaching the fictitious tier 1A 
status. Another moved from 19 to 16, all within tier 1A. That college was close to 
qualifying only in the sense that its starting point was close to the border between the two 
sub-tiers. Only one college made the reverse movement from tier 1A to tier IB. That 
institution moved from rank 18 in 1996 to rank 23 in 2000. One close call had movement 
17 to 18 during that period, but fell to 21 in 1999. 



Results and Discussion - Southern Regional Universities (Master’s 
Level) 

This part of the paper summarizes the results of a study of the rankings for the six 
years (1996, 1998-2002) 2 in the U.S. News and World Report (USNWR) for Southern 
Regional Universities-Master’s. Six years of rankings gave the possibility of five cross-tier 
movements: 1996-1998, 1998-1999, 1999-2000, 2000-2001, and 2001-2002. Universities 



2 The researchers were not able to obtain 1997 data. 
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could stay the same over all years or change. If they changed, they could change up or down 
repeatedly in several combinadons. 

If one looks at the probability of any changes taking place out of all opportunides for 
change to have occurred, the value turned out to be .19. In other words, change occurred in 
19 percent of the opportunides. Therefore most of the year-to-year comparisons showed no 
change. The probability of no change from one year to the next was 1.00 minus .19 or .81, 
i.e., 81 percent of the opportunides. The probability of a change (.19) was somewhat 
unevenly divided between rising and falling, with probability of reaching a higher der being 
.105 and that of declining being .086. 

Looking at the insdtudons, there were 143 represented over the six years. The 
breakdown was as follows: 111 insdtudons appeared over all six years, 11 insdtudons 
appeared five dmes, 1 insdtudon appeared four times, 2 insdtudons appeared two dmes, and 
18 insdtudons appeared once. 

Out of 111 insdtudons that appeared over all six years, 47 insdtudons showed no 
between-der movement at all over six appearances. The conservadsm is shown most of all 
in der 1 (20 insdtudons did not move), which is pardy due to a “ceiling effect.” 

Since the bottom 3 ders are not ranked, it is difficult to determine the average 
movement per insdtudon over the 5 years of possible change. As with the examinadon of 
the liberal arts category, such a study fell outside the resources of dme available for the 
current study. We used a shortcut to making a relevant esdmate. The shortcut was to 
calculate the average change in der 1 between 2000 and 2002. The average change turned 
out to be 2.3 posidons. 
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A finding of 2.3 positions is interesting because it represents only 6 percent of the 
range between ranks of 1 and 38. The greatest upward movement, from the 2000 to the 
2002 rankings, was 1 1 positions. The greatest downward movement was 9 positions. 

Movement over all 6 years (1996-2002) 

If movement happened similarly within the other three (unranked) tiers, we may 
expect that movement between tiers should be fairly rare. However, when considered over 
all 6 years, movement up and down was not particular rare. Considering the movement 
from tier 4 to tier 3, there were 41 institutions that had the opportunity to make that 
movement during at least one year during the period 1996-2002. Of these 41 institutions, 21 
moved up at least once. This number is equivalent to approximately 50 percent. 

However, even if movement was not particularly rare, one needs to know whether 
the overall upward movements tend to be balanced by downward movements. In other 
words, can the overall movement pattern be described as “zero sum”, with upward 
movement of some colleges balanced and compensated by the downward movement of 
other or (in many cases) the same colleges. One relevant observation was the following: 
while 21 institutions moved from tier 4 to tier 3, 21 also made the reverse shift from tier 3 to 
tier 4. 

There were 51 institutions in tier 3 that had the opportunity to move up to tier 2 at 
least once between 1996 and 2002. Of these, there were 18 that moved up at least once. 
Thus, such movement was not unusual. However, there were also 16 that moved down 
from tier 2 to tier 3 at least once. Many of the same institutions that moved up also moved 
down. 

There were 50 institutions in tier 2 that had the opportunity to move up to tier 1 at 
least once between 1996 and 2002. Of these, there were 16 that moved up at least once. 
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However, there were also 10 that moved down from tier 1 to tier 2 at least once between 
1996 and 2002. 

A Comparison of Movement Between the 1996 rankings and the 2002 rankings 

If we look at the movement of the 111 institutions that appeared both in 1996 and 
2002, 75 institutions did not change their tiers, 25 moved up, and 11 institutions moved 
down. 

There were 31 institutions that were in tier 4 who had, theoretically, the opportunity 
to move up between the 1996 and 2002 rankings. One university moved to a different 
category (National Universities). There were seven institutions that moved from tier 4 to tier 
3. There were two institutions that moved from tier 4 to tier 2. This number (i.e., 9 
institutions from 4-to-3 and 4-to-2) is equivalent to 30 percent or about 1 in 3. 

If membership within the Best Colleges list is conservative, changing only rarely, and 
a relatively small number rise in rank, it is possible that the overall movement pattern within 
the list can be best described as “zero sum”. That is, upward movement of some universities 
is balanced and compensated for by the downward movement of other universities. For 
example, while the seven universities moved from tier 4 to tier 3, six universities made the 
reverse trip from tier 3 to tier 4. 

Again there were 31 institutions in tier 3 that had, theoretically, the opportunity to 
move up between 1996 and 2002. Two universities moved to a different category (National 
Universities). One institution dropped from the USNWR rankings. There were six 
universities that moved up between 1996 and 2002. Further, there were three universities 
that moved down from tier 2 to tier 3 between 1996 and 2002. 

Again there were 31 institutions in tier 2 that had, theoretically, the opportunity to 
move up between 1996 and 2002. Three universities changed their categories; two joined 



the National Universities category, and one moved to the Comprehensive Colleges category. 
Ten universities moved from tier 2 to tier 1 between 1996 and 2002. Further, there were 
two universities that moved down from tier 1 to der 2 between 1996 and 2002. 

There were 32 institutions in tier 1. Six institutions changed their categories: three 
moved to the Liberal Arts Colleges category, two moved to the National Universities 
category, and one moved to the Comprehensive Colleges category. To further analyze 
movement in tier 1, we created an artificial division within tier 1 between institutions with a 
rank greater than 15 (i.e., 16-32) and those with a rank less than or equal to 15 (i.e., 1-15). 

Let us call them tier 1A (most premier) and tier IB (less premier). If we look for movement 
between the two new groups, we can measure the change in rank of those colleges that 
moved in either direction. It turns out that out of 24 institutions that stayed in tier 1 over all 
six years, only one institution moved from IB to 1A. And one institution made the reverse 
movement from tier 1A to IB. 

Implications and Future Research 

The information presented in this paper provides one important source of guidance 
for institutions wishing to improve their rankings. In general, there is sufficient stability 
within the rankings to make one take a somewhat skeptical view toward efforts to improve 
an institution’s rankings. At the same time, the same finding bodes well for those 
institutions that happen to be content with their rankings because they are as high as they 
can be or as high as college representatives expect them to become. Such institutions can 
afford perhaps to relax their efforts in regard to raising or maintaining their high status as far 
as these rankings are concerned. However, this general conclusion, while defensible in light 
of the results, should not be construed as discouraging the efforts of those institutions that 
would like to improve their rankings. Neither should the results be construed as offering an 
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excuse for complacency among those institutions that happen to be content with the 
rankings that they already possess. 

Before coming to those conclusions, several cautions should be observed. First, it is 
exceedingly important for an institution to determine its approximate rank within the tier to 
which it has been assigned. (For tier 1 institutions this information is provided and easily 
retrieved.) Without such information, it is impossible to determine whether efforts to 
improve are likely to bear fruit within less than 10 years. It could be argued that the 
institutions that showed the most movement (often multiple times, both up and down) had 
ranks that were close to the tier margins or borders. If so, the majority of institutions that 
did not move probably had a longer “distance” (in ranks) to move before a change in tier 
status would become manifest. By such reasoning, it appears important to know 
approximately where an institution ranks (i.e., in detail, not merely in its tier assignment) as 
useful information when attempting to raise an institution’s rank. Equally true, it is 
impossible to know without estimating those rankings whether the institution is at risk of 
moving backward in tier assignment or falling off of the list altogether. 

A second caution is that institutions would do well to reflect upon what they would 
do if the rankings did not exist at all. They should consider whether the efforts expended to 
improve or maintain rank or tier assignment are substantially the same as the efforts that 
would be expended if U . S. Neivs and World Report were not in this business. If the answer to 
that question is that the efforts would be substantially the same, then it might make better 
sense for the institution to use scarce resources to pursue the goal of improving its rank. 
Those efforts could more easily be defended to the degree that they merely duplicated the 
efforts that would be expended if there were no rankings to drive them. 
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Regarding future research, beyond suggesdons made above, we recommend the 



following program: 

1. Replicate the study for other groups/categories of insdtudons. 

2. Idendfy what market segments (classificadon, region) are the most compeddve 
(i.e., have higher probability of change). 

3. Utilizing six-year averages, compare movers and non-movers, and upward 
movers versus downward movers. Run discriminant analysis to determine the 
linear combination of independent variables (e.g., changes in spending patterns) 
which best discriminates between groups. Similarly, logistic regression can be 
utilized to contrast the groups. 
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FIGURE 1 



Frequencies at each number of appearances (1-6) of 162 
nationally-ranked liberal arts colleges and universities between 

the 1996 and 2001 rankings. 
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FIGURE 2 



Frequencies at each number of aDDearannes nf 143 cnnthom 
universities-master’s between the 1996 and 2002 rankinns 
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FIGURE 3 



Probability of tier change 
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FIGURE 4 



Mean Annual Rank Change in Tier 1 




□ Nationally-ranked Liberal Arts Colleges (1996-2000) D Universities-Master's (South) (2000-2002) 
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FIGURE 5 



Number of institutions that made at least one movement between tiers over 6 
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□ Nationally-ranked Liberal Arts Colleges (1 996-2001 ) E Universities-Master's (South) (1 996, 1 998-2002) 
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FIGURE 6 



Percentage of institutions that showed no movement at all over all 6 

appearances 




□ Nationally-ranked Liberal Arts Colleges (1 996-2001 ) □ Universities-Master's (South) (1 996, 1 998-2002) 
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FIGURE 7 



Percentage of the institutions that appeared in all 6 rankings and made 



movements 
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□ Nationally-ranked Liberal Arts Colleges (1996-2001) 

□ Universities-Master's (South) (1996, 1998-2002) 
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FIGURE 8 



Movement of 1 1 1 universities-master's (south), that appeared in all 6 years, 

between 1996 and 2002 
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FIGURE 9 



Number of u n iversities-master's (south) that changed tiers between 1996 and 

2002 
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